-
Notifications
You must be signed in to change notification settings - Fork 162
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add .NET Swift interop tooling components and layout #312
Add .NET Swift interop tooling components and layout #312
Conversation
…he flow with artifacts
Swift structs and enums can be projected into C# as either original types or C# classes. | ||
|
||
Some Swift structs are of scalar types which are essentially structs with one or more fields that contain identical blittable types, making them directly translatable to C# as structs. On the other hand, there are Swift structs that do not fit the scalar model. These non-scalar structs require representation in C# as IDisposable classes. The reason for using IDisposable is that Swift structs behave differently from C# value types. They have distinct semantics when they enter and exit scope, potentially involving changes to reference counts or destructors. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's worse than this. Value types in swift have specific semantics for what happens when an instance goes out of scope. This is, of course, completely different than the semantics of C#, but can be approximated by making the type IDisposable
. Early on in BTfS, I tried to classify structs into two types: blittable (contains 0 or more fields that are blittable) and non-blittable so that we handle them differently. Blittable types should be mappable to C# structs directly. What I found in reality is that there were so many edge cases that trying to do something with a little more efficiency in some cases created nothing but problems. Over and over again in BTfS, I was schooled that wherever possible, a general solution is best. This is why structs and non-trivial enums are best implemented as a common class implementing IDisposable
with a payload that is opaque.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I think the best way to model structs is either with an IDisposable
class or with a struct that contains an IDisposable
field that exposes the opaque pointer. To handle lifetimes, I would recommend that we emit a finalizer to ensure that we release memory correctly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, updated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the purpose of the IDisposable
implementation? Given that IDisposable
in C# doesn't actually indicate de-allocation, I would not expect it to do things like decrease a refcount on the Swift side.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is for disposal or release of native resources. In this case that is the intent. The fact that it is a refcount on the native side seems like implementation detail of the taret platform we are interoping with.
Swift structs and enums can be projected into C# as either original types or C# classes. | ||
|
||
Some Swift structs are of scalar types which are essentially structs with one or more fields that contain identical blittable types, making them directly translatable to C# as structs. On the other hand, there are Swift structs that do not fit the scalar model. These non-scalar structs require representation in C# as IDisposable classes. The reason for using IDisposable is that Swift structs behave differently from C# value types. They have distinct semantics when they enter and exit scope, potentially involving changes to reference counts or destructors. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I think the best way to model structs is either with an IDisposable
class or with a struct that contains an IDisposable
field that exposes the opaque pointer. To handle lifetimes, I would recommend that we emit a finalizer to ensure that we release memory correctly.
I have updated this PR according to the received feedback. The scope has been narrowed to focus on the general mechanisms of projections. For comprehensive details please refer to dotnet/runtimelab#2512. We've added validation steps for the projection tooling with required runtime support in dotnet/runtime#95636. Additionally, there are proposed stages for evolving the tooling in dotnet/runtime#95633. Feel free to get involved and share your perspectives or feedback on any provided issue. |
This PR should outline the general concepts of projection tooling. All details should be added to dotnet/runtimelab#2512. Please let me know if there is anything else that you think should be added here. |
Could there be a little more elaboration on the |
Good point. I've updated the docs to reflect the decision.
|
proposed/swift-interop.md
Outdated
Swift has a strongly-defined lifetime and ownership model. This model is specified in the Swift ABI and is similar to Objective-C's ARC (Automatic Reference Counting) system. When .NET calls into Swift, the .NET GC is responsible for managing all managed objects. Unmanaged objects from C# should either implement `IDisposable` or utilize a designated thin wrapper over the Swift memory allocator, currently accessible through the `NativeMemory` class, to explicitly release memory. It's important to ensure that when a Swift callee function allocates an "unsafe" or "raw" pointer types, such as UnsafeMutablePointer and UnsafeRawPointer, where explicit control over memory is needed, and the pointer is returned to .NET, the memory is not dereferenced after the call returns. Also, if a C# managed object is allocated in a callee function and returned to Swift, the .NET GC will eventually collect it, but Swift will keep track using ARC, which represents an invalid case and should be handled by projection tools. | ||
Swift has a strongly-defined lifetime and ownership model. This model is specified in the Swift ABI and is similar to Objective-C's ARC (Automatic Reference Counting) system. When .NET calls into Swift, the .NET GC is responsible for managing all managed objects. | ||
|
||
The `IDisposable` provides an explicit mechanism for releasing unmanaged resources. Destructors are managed by the GC and offer a way to release unmanaged resources when an object is collected by the GC. While destructors abstract away memory management from the user, the `Idisposable` pattern provides deterministic control over when resources and can lead to better performance as it prevents the need for GC collection cycles. The `IDisposable` pattern is the typical .NET approach for dealing with unmanaged resources and thus is selected as default option at initial stage. If it is determined that the `IDisposable` pattern introduces unnecessary overhead for users, and that destructors can adequately manage the release of unmanaged resources, appropriate updates to the memory management approach will be made. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the Idisposable pattern provides deterministic control over when resources and can lead to better performance as it prevents the need for GC collection cycles
How is this relevant to interop? It sounds like it doesn’t make any semantic difference if the object is released early or late, it’s just a design difference. That is, it doesn’t sound like there are any non-memory resources being tracked. In that case I would expect us to stick with C# design, which is to use the GC for all memory resources. Is there something else being tracked here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there something else being tracked here?
Composition can be a concern here. For example, in COM Aggregation there are potentially two distinct memory models that are represented by a single object - WinRT/COM class is the base of some .NET class. In the COM scenario this is relatively simple because it is expected the user to handle circular references that strattle the interop boundary. For WinRT, the Jupiter runtime has a reference tracking mechansim that means more cooperation is needed to collect a single object that is managed by the two systems.
I don't know if our Swift interop scenarios will have need for that sort of mechanism, but from experience interop scenarios that are non-deterministic make tooling very complicated - memory tracking, code coverage, etc. If we expect existing Swift tooling to be usable when .NET is involved having an explicit option is beneficial - even if it is an opt-in like CreateObjectFlags.UniqueInstance
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The primary reason for considering destructors is to utilize the GC and the C# design pattern. The current tooling (BTfS) implements the IDisposable
strategy. Since we are at an early stage, it is challenging to make a decision.
I don't know if our Swift interop scenarios will have need for that sort of mechanism, but from experience interop scenarios that are non-deterministic make tooling very complicated
Opting for IDisposable
for the sake of simplicity seems like a reasonable argument. @agocke are you open to this approach (starting with IDisposable
), or do you prefer starting with destructors initially and transitioning to IDisposable
if required?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My concern is mainly that we overload the use of IDisposable
a lot in .NET, and that makes it confusing for customers. In some cases, like for files, it's pretty important that you call IDisposable
to prevent running out of handles. In this case, pretty much every Swift type would come with IDisposable
. Without knowing details of the type, it seems impossible for me, the caller, to understand whether I should or shouldn't dispose it.
Moreover, I think the preferred behavior for "trivial" deinit calls would be to not call Dispose (or more accurately, not wrap the type in a using
). Since adding a using
in C# produces a try-finally and hurts codegen, it's probably better to avoid a pattern that causes an explosion of usings. Instead, I think it would be ideal if we only used IDisposable for custom deinit. In those cases, we can be relatively confident that something more than just memory is being freed.
So overall I'm very supportive of some way to explicitly call deinit/dispose on Swift type, but making everything IDisposable feels like an anti-pattern that will just cause a lot of user confusion.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for info. I've updated the section and the MVP accordingly.
|
||
The Binding Tools for Swift tooling handles these explicit lifetime semantics with generated Swift code. In the new Swift/.NET interop, management of these lifetime semantics will be done by the Swift projection layer and not by the raw calling-convention support. If any GC interaction is required to handle the lifetime semantics correctly, we should take an approach more similar to the `ComWrappers` support (higher-level, less complex interop interface) rather than the Objective-C interop support (lower-level, basically only usable by the ObjCRuntime implementation). | ||
There are two strategies for managing native memory in .NET: destructors and `IDisposable`. Destructors abstract away memory management from the user and are managed by the GC. They provide a way to release unmanaged resources when an object is collected by the GC. `IDisposable` offers an explicit mechanism for releasing unmanaged resources with deterministic control over when resources are released. The preferred behavior for general cases would be to implement destructors. This approach aligns with the .NET pattern and offers codegen benefits by avoiding excessive `using` statements. Ideally, the tooling should only use `IDisposable` for custom deinit. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What are the compatibility rules for adding/removing custom deinit in Swift?
Are introduction or removal of a custom deinit considered to be a compatible change in Swift? If they are compatible changes, it would be problematic to map custom deinit to IDIsposable. Adding/removing IDisposable is a breaking change in .NET.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is unfortunate -- I bet Swift is fine with removing a deinit since there's always an implicit deinit that it can fall back on.
I think the fundamental problem here is that Swift does not have a distinction between "deterministic" destruction and non-deterministic destruction. In contrast, C# does. And, importantly, that distinction is very important in C#. If you require deterministic destruction then timely destruction may be a requirement for your application, where delaying destruction may cause your application to fail. Conversely, if you don't require it, using deterministic destruction (IDisposable
/using) for all resources is very expensive and will likely cause significant performance degradation. It's also a large programmer burden as disposability is transitive and any IDisposable fields require the parent type to also be IDisposable.
"Custom deinit" is a heuristic that attempts to map between C#, which differentiates the two concepts, and Swift, which doesn't. But since there is no way to determine statically whether a Swift type actually requires deterministic destruction, I don't think there's any heuristic that will be perfectly accurate.
That said, maybe we can do better than "custom deinit". Suggestions welcome.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this imply that we should have a solid reason for implementing the IDisposable
pattern on a type as introducing or removing explicit memory management via IDisposable
could lead to breaking changes?
This is the case where direct mapping is not possible, and introducing IDisposable
only when absolutely necessary can help minimize breaking changes caused by switching to/from the explicit memory management.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the case where direct mapping is not possible, and introducing IDisposable only when absolutely necessary
Another option here would be to follow the built-in COM mechanism and add a helper function like Marshal.FinalReleaseComObject()
. This could be on Marshal
or another more targeted type, and use any number of tricks to provide efficient deterministic release semantics. This has the added benefit of not forcing users to litter their code with type checks/casts.
Note that ComWrappers
did use IDisposable
for specific COM scenarios. In that case though the user is driving the experience by explicitly requesting the deterministic support - see this section of the ComWrappers
tutorial.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll offer a datapoint here that there are at least 4 issues at play here:
- Swift has very different semantics than C# for value types. Value types can induce code to get executed when they go out of scope (for example, a value type contains a reference type).
- Swift maintains two different reference counts: a strong reference count and a weak reference count on heap-allocated types, not a simple reference count. The weak reference count is always used implicitly at allocation time (IIRC - it's been a while since I put this under a magnifying glass) and may be used explicitly by assigning to a member declared as
weak
- At binding time, it is not always deterministic that a value type will always be blittable (for example, a non-frozen struct could get a member added in the future) and this will create a compatibility issue for us in the future
In BTfS, when we are presented with a Swift object, the C# binding implements IDisposable
and takes a swift weak reference to the object via the swift runtime routine swift_unownedRetain
and keep a weak GCHandle. Dispose (false)
gets called by the finalizer.
Initially, I tried to draw a distinction between blittable and non-blittable value types, but things that make that challenging were all the special cases: value types that contain mutating methods or properties, inout
parameters, frozen/not frozen, opaque layout of enums etc. It was an endless supply of bugs in the marshaling. My solution for this was to make all values type handling uniform.
Not being an expert on it, does COM interop exist on non-windows platforms?
I'm pinging @rolfbjarne here as well since he was responsible for the handling of ObjC mapping, which required similar work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's review some examples of existing bindings to gain a deeper understanding of how the IDisposable
pattern is utilized. I've done a review of the Xamarin Google APIs for iOS Components and Google Mobile Unity Ads, focusing specifically on the following applications: cloud messaging, mobile ads, analytics, and crashlytics. The goal of this review was to identify the use cases of IDisposable patterns. The collected data could help us move this discussion/decision forward.
In the existing Xamarin ObjC interop, the NSObject
represents a base object and implements IDisposable.
Cloud Messaging
In the cloud messaging bindings, the following APIs require implementing the IDisposable pattern:
MessageInfo
MessagingDelegate
Messaging
MessagingExtensionHelper
In the sample application, these types are used in AppDelegate, but there are no explict disposing of objects.
There is another example where IDisposable pattern is used to remove notification handles
var sendSuccessToken = Messaging.Notifications.ObserveSendSuccess (HandleSendMessageSuccess);
var sendErrorToken = Messaging.Notifications.ObserveSendError (HandleSendMessageError);
// Call this lines to stop receiving notifications
sendSuccessToken.Dispose ();
sendErrorToken.Dispose ();
// Another option: Calling native code could explicitly release observers.
var sendSuccessToken = NSNotificationCenter.DefaultCenter.AddObserver (Messaging.SendSuccessNotification, HandleSendMessageSuccess);
var sendErrorToken = NSNotificationCenter.DefaultCenter.AddObserver (Messaging.SendSuccessNotification, HandleSendMessageError);
// Call this lines to stop receiving notifications
NSNotificationCenter.DefaultCenter.RemoveObserver (sendSuccessToken);
NSNotificationCenter.DefaultCenter.RemoveObserver (sendErrorToken);
Analytics
In the analytics bindings, the following APIs require implementing the IDisposable pattern:
Analytics
DictionaryBuilder
Fields
Logger
TrackedViewController
Tracker
EcommerceFields
EcommerceProduct
EcommerceProductAction
EcommercePromotion
In the samples, they are not explicitly released.
MobileAds
In the mobile ads bindings, there are 43 interfaces with base types implementing IDisposable section.
There are 6 instances of disposal in the sample application:
// You need to explicitly Dispose Interstitial when you dont need it anymore
// to avoid crashes if pending request are in progress
void RemoveAdFromTableView ()
{
if (adViewTableView != null) {
if (adOnTable) {
dvcDialog.Root.RemoveAt (idx: 2, anim: UITableViewRowAnimation.Fade);
}
adOnTable = false;
// You need to explicitly Dispose BannerView when you dont need it anymore
// to avoid crashes if pending request are in progress
adViewTableView.Dispose();
adViewTableView = null;
}
}
GoogleAds Mobile Unity
In the google mobile bindings, the following types implement IDisposable, but none of them are disposed explicitly.
AdManagerBannerClient
AppOpenAdClient
BannerClient
InterstitialClient
NativeOverlayAdClient
RewardedAdClient
RewardedInterstitialAdClient
Crashlytics
In the crashlytics bindings, the following APIs require implementing IDisposable:
Crashlytics
ExceptionModel
StackFrame
In the sample application, none of them are disposed explicitly.
Xamarin Designer
The Xamarin Designer (deprecated) for iOS is a visual designer that contain controls with IDisposable pattern. Samples from the mentioned bindings that implement Xamarin Designer templates contain a ReleaseDesignerOutlets function that explicitly releases all unmanaged references. Here is an example:
[Outlet]
UIKit.UILabel LblPreview { get; set; }
[Outlet]
UIKit.UILabel LblSubtitle { get; set; }
[Outlet]
UIKit.UILabel LblTitle { get; set; }
void ReleaseDesignerOutlets ()
{
if (LblPreview != null) {
LblPreview.Dispose ();
LblPreview = null;
}
if (LblSubtitle != null) {
LblSubtitle.Dispose ();
LblSubtitle = null;
}
if (LblTitle != null) {
LblTitle.Dispose ();
LblTitle = null;
}
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @kotlarmilos, this is a lot of great work!
Based on real-world examples, it doesn't seem like people actually use Dispose in most cases. Based on that, I would recommend using a static method like Marshal.ReleaseComObject (or even a different interface if necessary) rather than adding a bunch of ceremony to the API, but that's just my opinion.
My main concern is accurately providing guidance to authors on whether or not manual disposal is expected, and it seems like it's not.
@jkoritzinsky @AaronRobinsonMSFT @jkotas thoughts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @kotlarmilos, this is a lot of great work!
+1. Thank you for this sort of due-diligence and investigation.
I would recommend using a static method like Marshal.ReleaseComObject (or even a different interface if necessary) rather than adding a bunch of ceremony to the API
Agree. I accept this is anecdotal, but it does paint a compelling argument for avoiding a ubiquitious IDisposable
requirement. I think the onus is now on the other side of the argument to make a case for IDisposable
everywhere.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 on Andy's and Aaron's responses - and thanks @kotlarmilos for the analysis.
Just curious - did we consider adding an instance method onto the base type (assuming there will be a base type for all projected types)? I know this is sort of against .NET/C#, but I might go as far as adding a void Dispose()
onto the base type without implementing IDisposable
. Unfortunately C#'s using
is not pattern based (unlike most other C# features) and needs the type to implement IDisposable
. But it would make the disposing feel a bit more first class and more discoverable I think.
Additional notes:
- Implementing
IDisposable
on the projected types can trigger analyzer warnings- for example CA2000, or CA2213 - although these analyzers are not enabled by default currently (that said it seems that other tools do report similar diagnostics - Resharper and so on) - I do agree that for certain projected types implementing
IDisposable
would be a good idea (like the above mentioned photos) - this means that eventually the tooling should be able to take a hint to implement the interface on a given projected type. - Based on the above description it seems that the cases where
Dispose
is called are actually not about releasing resource pressure, but about actual functionality - like de-registering something. It just so happens that the semantics is tied to "deinit" because in the Swift that's the right design. So requiring it on all types feels weird...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Something like Dispose
is fine but I might be careful about using Dispose
itself because it could be confused with IDisposable and raise the same problems. Not implementing the interface is a kind of subtle shift that doesn't seem to present a lot of clarity.
Implementing IDisposable on the projected types can trigger analyzer warnings- for example CA2000, or CA2213 - although these analyzers are not enabled by default currently (that said it seems that other tools do report similar diagnostics - Resharper and so on)
This is indeed my main concern -- some IDisposable
types require disposal for correct usage. The Swift types don't. It's really hard to tell general IDisposable
implementations that do require Dispose from the ones that don't, so my general recommendation for people is to avoid IDisposable
if it's not intended to be used in normal operation.
Closing this PR as it is obsolete. The feedback will be incorporated into the design of the tooling. |
Description
This is the initial PR for the projection tooling which introduces the .NET Swift interop tooling components based on the Binding Tools for Swift and highlights the functionalities of the tool. Subsequent PRs will cover the process of projecting Swift types into .NET.
For importing Swift into .NET, the key components are presented, where they work together to allow the generation of C# bindings and Swift wrappers for interop. The process begins with the source Swift library, from which a module declaration is generated. Subsequently, the tool generates Swift wrappers and its module declaration for cases where direct P/Invoke is not possible. Finally, the tool generates C# bindings. For exporting .NET to Swift, the tool generates corresponding Swift and C# wrappers to enable type mapping.
Regarding distribution and validation, the tooling will be part of Xamarin publishing, shipped as NuGet package and accessible via .NET CLI. Validation will be achieved through supporting libraries and MAUI samples, as outlined in dotnet/runtime#95636.